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> EFB authorization process 

> EFB authorization and AQP 



> AGP Small Group Tryout 

> Instructor/Evaluator training and standardization 

> Use of data to pinpoint areas for improvements 

> Value of longitudinal data collection and analysis. 
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EFB Authorization 



EFB authorization includes five phases: 

> Phase One: Initiation (2 to 1 0 weeks) 

> Phase Two: Application (8 to IS weeks) 

> Phase Three: Principal Inspector (PI) Review (4-8 
weeks) 


> Phase Four: Interim Authorization to Use an EFB 
(possibly several years) 



Phase Five: Authorization to Use an 
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Phase Two 


6th week 


8th week 
9th week 
10th week 
11th week 
12th week 


13th vi 
14 th vi 
16th y 


lath v 

19th v 


20th v 
21st w 


Operator Activity/Product 


Phase One: Initiation 


Finalize STC. TC and authorization responsibilities 
with EFB suppliers and installers 


Develop EFE Implementation Timeline, form EFB 
team and complete EFB Tabletop Evaluation 


Operator internal meeting with engineering, 
standards, training, AQP, dispatch and maintenance 


Review procedures and documents to identity EFB 
additionsJchangea 


Prepare and conduct internal review of Request for 
Authorization and Outline Format EFB Plan 


Submit Req ues f for Authorization & Outline 
^omTa^^^^ar^jeootiatin^an^change^witt^^^ 
Phase Two: Application 
Complete and submit Formal EFB Plan to PI and 
start procedures, documents and training revisions 
Prepare Draft EFB related documents 
lAFWFOM/MEL/ checklrst> addition s/changes 
Negotiate final agreement with PI on Formal EFB 
Plan with any changes in writing 
Complete Draft EFB pilot /instructor training lessons 
and crew assessment procedures 
Complete Small Group Trial of EFB procedures 
training and performance/proficiency (this may be 
optional but advisable especially for an AQP fleet) 
Prepare EFB Evaluation Report 
Assemble and review EFB Application Package 
Submit EFB Application Package to PI and Small 
^Grou^^a^esutt^t^AQ^i^n^Qf^leet^^^^^^ 


{see Order 8900.1 , Vd. 
Ch. 15. Figure 4-75) 


{see Order 390D.1, VoJ. 
4. Ch. IS. Figure 4-79) 


Specify timeline 

and products 


Identify ACQ . AEG , and 
AQP requirements 


Include pilot groups 


Specify ACO, AEG and 
AQP contacts 


Check Appendix A-F of 
fhis Guide 
See Table 2 

Clarify any changes to 
Formal EFB Plan 

See Table 4 

Involve PI and pilot 
groups in trial 

See Appendix D 

Include pilot groups 


Phase Throe: PI Review 


Install EFB training devices (classrooms andfor 
simulelor} and trai n l/Ea 


Negotiate any changes to the EFB Applies lion 
Package with PI 


Review' EFB procedures, documents and training 
packages 


Ensure EFB deta collection ii 
ready for implementation 


comprehensive and 


3 months or more 


Phase Four: Interim Authorization 


Ensure that l(Es use 
slanderd assessment 


Include PI end pilot 
grou os 


Implement date collection for the EFB Validation 
Final Report 


Evaluate and refine data collection For fine EFB 
Validation Final Report 


Prepare, review and submit EFB Valida tion Final 

Report to PI 

F vc 2 i ’ l '! o ■ izjnoi 


See Appendix F 


9 months or more 


I D months or more 


Ensure authorization for use is placed ii 
AQ25 


Monitor and report ongoing EFB crew 
performBnce/proficiency end EFB reliability, and 
analyze 1o refine procedu restraining . 



Phase Two Operator Steps 

Step 2.1 Use Outline Formal EFB Plan prepared in Phase One to complete the Formal 
EFB Plan and submit it to the PI agreeing on a date when the PI will render a 
decision. 

Step 2.2 While the Formal EFB Plan is being reviewed by the PI, start work on the changes 

to Existing Operating Documonts 

• AFM and FOM . initially released as a bulletin, providing conditions, limitations, 
philosophy and procedures for the use of the EFB 

• Checklists to include EFB checking, startup and shutdown 

• Flight Training Manual including description of EFB system components, 
functionalities, differences, and usage under a vanety of operational 
conditions for alt flight phases 

• MEL including the EFB system in accordance with the MM EL 

• General Maintenance Manual including on-going maintenance and inspection 
of EFB equipment. 

NOTE: Use the checklist in Table 3 to further identify all documents and sections 
that might require new or revised EFB information 

Step 2.3 Review Formal EFB Plan with PI and obtain a written agreement of the Plans 
acceptance along with any required changes. 

Step 2.4 Work with EFB suppliers) to gather all required documents specified in the 

Formal EFB Plan pertaining to EFB Crashworthiness. Rapid decompression and 
Non-interference 

Step 2.5 Review results from Step 1.3 (EFB Tabletop Evaluation) and finalize EFB 

procedures and training to address any unresolved issues. 

Step 2.6 Finalize changes to Existing Operating Documents and have pilots review the 
EFB system completing Items 1-91 from Checklist 2, EFB Operational Evaluation 
(see Order 8900 I.Vol. 4. Ch 15 . Figure 4-80). 

Step 2.7 Use Items 92- 107 in Checklist 2, EFB Operational Evaluation {see Order 8900.1, 
Vol 4, Ch. 15. Figure 4-80). to develop an EFB Line Oriented Evaluation (LOE) 
and conduct a small group trial if possible at this time. A small group trial may be 
performed using EFBs installed in a simulator with 4-8 crews flying the LOE If a 
small group trial is possible, use its results to complete Checklist 2 - EFB 
Operational Evaluation, Items 92-109. 

NOTE: If a small group trial is not possible dunng Phase Two. specify which form 
of EFB Operational Evaluation will be conducted by Phase Four. 

Step 2.8 Prepare the EFB Evaluation Report which at a minimum includes a cover page 
and the completed Checklist 1 (see Step 1 1) and Checklist 2 (see Step 2.6) with 
supporting data. The cover page should provide an explanation of items that were 
marked as No in either Checklist 1 or 2 (e.g., the item is not applicable, or the item 
will be addressed by Phase Four, etc ). See Appendix D for sample outline. 

Step 2.9 Review the EFB Application Package for completeness using the checklist in 

Table- 3 and inen submit it to the PI. 
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Two; 



Phase Two includes: 

> Existing Operating Documents: Make changes 
required for EFB implementation 

> EFB Supplier Documents: Obtain required tests 

> EFB Procedures & Training: Modify using EFB 
Tabletop evaluation results 

> Small Group Tryout: Collect data and prepare 
Evaluation Report 



EFB Application Package: Complete and submit. 
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The EFB Evaluation Report requires the 


evaluation of the EFB 




& 

t 


> EFB Hardware 

> General User Interface 


> Software Applications 

> EFB Procedures 


> EFB T raining 

> Crew Performance (main phases of flight). 
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EFB Training and 






■: 



> Suitability of Normal Procedures 

> Suitability of Non-Normal Procedures 

> EFB Software Applications Training 

> Crew Workload 


> Currency for Proficiency. 
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Small Group Tryout (SGT) for EFB based on 
AQP guidance includes: 

> Prepare training and LOS materials 

> Train Instructors and Evaluators (!/Es) 

> Conduct approved curriculum 

> Analyze and submit performance/profidency data 

> Revise EFB training program. 
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Train l/Esi LOS Administration 


For this example we will consider a LOS 





> Event Set 1 : Preflight with several deferrals 

> Event Set 2: Push back with Generator Fail 


> Event Set 3; Climb with clearance change 

> Event Set 4; Hold and airport shut down 

> Event Set 5: Reroute with fuel considerations. 
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Train l/Es:, LOS Administration 


LOFT Event Set 1 

Conditions 

• APU GENERATOR AND LEFT CLEARVIEW WINDOW DEFOG DEFERRED 

• RELEASED VFR WEATHER 

• UNUSUAL DEPARTURE WITH RESTRICTIONS 

• TRWs ENROUTE 

Trigger 

• Receipt of PDC changes SID from Orlando 8 to Jaguar 3 


Event Set 1 Start: Receipt of PDC 

Time: 

Event Set 1 Stop: Crew updates FMS and completes 
cross-check using navigation publications. 

Time: 


Superior ■ 4 

Good — 3 

Problem = 2 

Repeat = 1 

CAPT use of FMS and navigational 
pubs to conduct brief 





FO setup and use of navigation 
pubs up to and through brief 






Analyzing AQP Data to Improve EFB Operations and Training, June 2, 2010 


11 




Shared Vision of Aviation Safety Conference 


.Train !/£s; Performance Assessment 

The eight l/Es in this example all viewed the same 
video of an average crew flying the LOS and after 
each event set were given several minutes to 

individually: 


> Note the start and stop time for each Event Set 

> Rate each Event Set independently 

> Write comments about the LOS administration,, 
simulator problems and any other issues. 
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Data was collected from all eight l/Es for the 18 
Captain and F/O behaviors being rated on the 
following four-point scale: 

>4 = Superior (substantially above standard) 

>3 = Good (standard crew performance) 

> 2 = Problem (problem that was resolved during LOS) 

> 1 = Repeat (one or more problems not resolved) 
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Of the numerous methods for Inter-Rater 



^Systematic Differences: Analysis of each l/E’s ratings 
compared to the group using a t test to identify those 
who are significantly different (< .05) from the group. 

> Agreement; Analysis of each behavior being rated using 
the r wg formula, and reviewing behaviors with a relatively 

low agreement with an r wq less than .5. 
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EFB l/E Check 

lffi-1 

VE-2 

l‘E-3 

HE-4 

HfE-5 

UE-G | 

| \IE-7 

HE-S 

ES1-GAPTFM5 & BrieF 

4 

3 

3 

4 

3 

2 

4 

3 

ES1-FO FME Support Brier 

4 

3 

2 

3 

2 

1 

3 

3 

ES2-CAPT MEL 

4 

4 

3 

4 

3 

3 

3 

3 

ES2-CAPT Decision 

4 

3 

2 

3 

2 

2 

3 

3 

ES2-FO MEL 

3 

2 

2 

2 

1 

3 

2 

2 

ES2-FO Support 

3 

3 

1 

2 

1 

3 

2 

2 

ES3-CAPT ID JS5 

3 

3 

2 

1 

2 

2 

3 

2 

ES3-FO ID JB5 

4 

3 

2 

2 

2 

3 

3 

3 

E33-CAPT FMS 

4 

3 

2 

3 

2 

3 

3 

3 

ES3-FO FMS 

4 

4 

2 

3 

2 

3 

3 

3 

ES4-CAPT FMS 

4 

3 

2 

4 

3 

3 

4 

3 

ES4-FO FMS 

3 

3 

1 

3 

1 

3 

2 

4 

ES4-CAPT Mav Pubs 

3 

3 

2 

2 

2 

2 

2 

2 

ES4-FO Mav Pubs 

4 

3 

1 

4 

2 

2 

3 

3 

ES5-GAPT FMS 

2 

3 

3 

2 

2 

1 

3 

2 

ES5-CAPT Decision 

3 

3 

2 

2 

2 

2 

2 

4 

ES5-FO EFB 

3 

3 

2 

2 

2 

2 

3 

2 

ES5-FO ATC Compliance 

4 

3 

2 

4 

1 

2 

2 

3 

Variance 

0.3B 

0 17 

0.35 

0.09 

0.41 

0,47 

0.42 

0.42 

Individual Average 

3.50 

3.00 

2.00 

2*76 

1.94 

2*35 

2*78 

2.78 

Group Average 

2.65 

2.65 

2 65 

2*65 

2 65 

2,65 

2,65 

2,65 

TTEST 

0.00 

0 Q0 

0.00 

0.42 

0 DO 

0,07 

0,25 

0,33 


HIGH 

HIGH 

LOW 

OKAY 

LOW 

OKAY OKAY OKAY 
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For this analysis, examine the data by l/E, looking 
at the mean and variance in each column to 
determine whether: 

>The Individual l/E ratings are significantly different from 
the group mean ratings: 

• If t test > than .05, then l/E is OK (l/E not significantly different from Group) 

• If f test < than .05, determine whether l/E is HIGH or LOW 
compared to the Group Mean. 
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l/E-1 

l/E-2 

l/E-3 

l/E-4 

l/E-5 

l/E-6 

l/E-7 

l/E-8 

Variance 

0.38 

0.17 

0.35 

0.89 

0.41 

0.47 

0.42 

0.42 

Individual Mean 

3.50 

3.06 

2.00 

2.78 

1.94 

2.33 

2.78 

2.78 

Group Mean 

2.65 

2.65 

2.65 

2.65 

2.65 

2.65 

2.65 

2.65 

t test Significance 

0.00 

0.00 

0.00 

0.42 

0.00 

0.07 

0.25 

0.33 


HIGH 

HIGH 

LOW 

OK 

LOW 

OK 

OK 

OK 


Note: This data was generated to emphasize specific points. 
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T rain l/Es: 




row to determine whether: 


> The specific item had acceptable agreement: 

• If rwg is £ than .6, then agreement is OK 

• If rwg is < than .5, then agreement should be reviewed 

> There is a pattern across the items showing low 
agreement 
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Train j/Es; Agjeemtnt(r wg >6 ok <.s problem) 


EFE l/E Chrck 

WE-1 

L'E-2 

HE-3 

l/E-4 

L'E-5 

l.'E-G | 

\lE-7 | 


AGREE 

ESI -GAFT F M3 & BrieF 

4 

3 

3 

4 

3 

2 

4 

3 

0.60 

ES1-FQFMS& Support Brie 

4 

3 

2 

3 

2 

1 

3 

3 

0,33 

ES2-CAPT MEL 

4 

4 

3 

4 

3 

3 

3 

3 

0.79 

ES2-CAPT Decision 

4 

3 

2 

3 

2 

2 

3 

3 

0,60 

ES2-FO MEL 

3 

2 

2 

2 

1 

3 

2 

2 

0.67 

ES2-FO Support 

3 

3 

1 

2 

1 

3 

2 

2 

0,44 

ES3-CAPT ID JS5 

3 

3 

2 

1 

2 

2 

3 

2 

0.60 

ES3-FO ID JB5 

4 

3 

2 

2 

2 

3 

3 

3 

0,60 

ES3-CART FM3 

4 

3 

2 

3 

2 

3 

3 

3 

0,6? 

ES3-FO FMS 

4 

4 

2 

3 

2 

3 

3 

3 

0,64 

ES4-CAPT FMS 

4 

3 

2 

4 

3 

3 

4 

3 

0.60 

ES4-FO FMS 

3 

3 

1 

3 

1 

3 

2 

4 

0,09 

ES4-CAPT N av Pubs 

3 

3 

2 

2 

2 

2 

2 

2 

0.83 

ES4-FO MavPubs 

4 

3 

1 

4 

2 

2 

3 

3 

0.14 

ES5-CAPT FMS 

2 

3 

3 

2 

2 

1 

3 

2 

0.60 

ES5-CAPT Decision 

3 

3 

2 

2 

2 

2 

2 

4 

0,64 

ES5-FO EFB 

3 

3 

2 

2 

2 

2 

3 

2 

0.79 

ES5-FO ATC Compliance 

4 

3 

2 

4 

1 

2 

2 

3 

0.10 
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Train l/Esi Standardize Assessment 

If l/Es show Systematic Differences or there is lack 

of Agreement ©n specific behaviors: 

> Work with l/Es as a team to discuss problems 

> Review the rating scales and markers as a group 

> Review relevant procedures and standards 

> Review LOS Event Sets and their assessment 

> Form a group consensus on pilot and crew assessment 

> Have l/Es rate and review until standardized. 


Analyzing AQP Data to Improve EFB Operations and Training, June 2, 2010 


20 



Shared Vision of Aviation Safety Conference 



Analyzing AQP Data to Improve EFB Operations and Training, June 2, 2010 


21 







Shared Vision of Aviation Safety Conference 



Curriculum 


Home Study 

> Legends and Tables document 

> Aircraft Bulletin 

> Website Operations 

> Review the publications on the website 

> Open book guided evaluation 
Classroom TraMrrgi 

> EFB hardware overview 

> Utilization of EFB and contingencies 

> Practice 

> Classroom quiz 
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Cumculum. 


Home Study 


Classroom Training 


Flight 

Manual 

Bulletin 



Electronic Route Manual Chart Clip 
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Conduc t EFB Curriculum 

Simulator LOS: 

> Crew LOS briefing 

> Crew flight planning 

> Operational Evaluation Simulator LOS 

> Crew members complete Workload Rating Form, 

Debrief: 

> Crew members complete Exit Questionnaire and 
debriefing. 
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Conduct EFB Curriculum; Data Collection, 

For the Small Group Tryout, consider the following: 

> Do not start collecting data until 1/E assessment of 
crews is standardized 

> Collect data on EFB as well as existing system usage 

> Use from 6 to 10 crews for each system evaluated 

> Ensure uniform LOS administration and data collection 

> Consider collecting situation awareness, head-down 
time and workload data in addition to crew performance 
data. 
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Looking at the eight l/Es, based on a Small Group Tryout of 
24 LOSs, note that l/E-2 still has a relatively low variance, 
and a check on the data shows that ll/E-2 is still most 
comfortable giving a rating of 3. Some of the other rater 
biases persisted, like l/E-1 being higher than the group. 



l/E-1 

l/E-2 

l/E-3 

l/E -4 

l/E-5 

l/E-6 

l/E-7 

l/E-8 

Variance 

0.36 

0.15 

0.34 

0.76 

0.39 

0.45 

0.40 

0.40 

Individual Mean 

3,41 

3.00 

2,00 

2.91 

1.94 

2.33 

2,78 

2,78 

Group Mean 

2.64 

2.64 

2.64 

2.64 

2.64 

2.64 

2.64 

2.64 


Note: This data was generated to emphasize specific points. 
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Based on l/E means and variance for the Small Group 
Tryout data, the following revisions should be considered 

> Review behavioral markers and the use of the 4-point 
rating scale 


> 

> 


Review the LOS Event Sets with an emphasis on EFB 
related conditions and triggers 

Provide additional I/E IRR training based on a group 
review of Small Group Tryout data. 
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ES1-CAPT FMS & Brief 

3.2 

ES1-FO FMS & Support Brief 

2.4 

ES2-CAPT MEL 

3.3 

ES2-CAPT Decision 

2.8 

ES2-FO MEL 

2.6 

ES2-FO Support 

2.6 

ES3-CAPT ID J85 

2.3 

ES3-FO ID J85 

2.4 

ES3-CAPT FMS 

2.8 

ES3-FO FMS 

2.4 

ES4-CAPT FMS 

3.3 

ES4-FO FMS 

2.4 

ES4-CAPT Nav Pubs 

2.6 

ES4-FO Nav Pubs 

2.7 

ES5-CAPT FMS 

2.6 

ES5-CAPT Decision 

2.5 

ES5-FO EFB 

2.4 

ES5-FO ATC Compliance 

2.6 


Note: This data was generated 
to emphasize specific points. 
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and Training 


Based on the overall crew performance Means of the Small 
Group Tryout, consider: 

> Reviewing and refining FO procedures, especially in the 
area of FMS data entry, crosschecking and monitoring 

> Identifying best practices of top performing crews 

> Refining the EFB training by adding best practices and 
showing how to use the EFB to crosscheck FMS data 

Refining EFB training in the area of crew coordination 
and workload management. 


Analyzing AQP Data to Improve EFB Operations and Training, June 2, 2010 


29 



Shared Vision of Aviation Safety Conference 


EFB Training, and. Best£ractices 

Sample EFB workload management best practices include: 

> Using the EFB Load Setup function during Preflight to 
reduce workload 

> Using the EFB Send function in ways to reduce crew 
workload 

> Setting up the approach charts and documents during 
times of low workload 

Using the EFB Print function in preparation for periods 
of heavier workload. 
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PreFlight Eng Start Taxi-Out-In Takeoff CLMB/DSC Cruise Landing 


□ Best PracticesP Standard Operating Procedures 
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> Initiating data collection and analysis early in the implementation 
process 

> Using a de-identifled system to better analyze longitudinal data 

> Using longitudinal l/E data to improve their standardization 

> Identifying above average pilots and crews and use their 
performance to specify best practices 

> Identifying problems before they become significant issues 

> Using similar Event Sets to evaluate crew performance 
using EFB compared with them using the old system. 
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Qrew Performance with E FB & Paper 

6.00 

5.50 

5.00 
g 4.50 
n? 4.00 

tu 

S 3.50 

i 300 

£ 2.50 

2.00 

1.50 
1.00 



Event 1 
FMS 


Event 1 
Briefing 


Event 2 Event 2 Event 3 
MEL Resolution SOP 


□ Electronic □ Paper 


• Yellow boxes indicate significant differences < .05 
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> Controller-Pilot Data Link Communications (CPDLC) 

> Cockpit Display of Traffic Information (CDTI) 
Automatic Dependent Surveillance-Broadcast, 
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Key Points. 

Initiate data collection and analysis early in the implementation 
process 

> Use data to identity procedural and training refinements 

> Use a de-identified system to analyze longitudinal data 

> Use longitudinal l/E data to improve their standardization. 

> Identity above average pilots and crews and use their 
performance to specify best practices 

> Analyze below average crew performance data to isolate 
problems with the training, evaluator standardization and pilot 
proficiency. 
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For Questions: 

Barbas Kanki 
barbara.g.kanki@nasa.gov 


Thomas Seamster 
tseamster@q west. net 
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